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ABSTRACT 

key concepts from theory and research on student 
learning are discussed, and a conceptual framework to promote the 
design of local instructional improvement efforts is presented. The 
objective is to advocate that institutional researchers focus more 
directly on the basic issues of teaching and learning and to suggest 
that improving instructional program quality will improve student 
learning, the framework identifies factors, or interveiiors, that 
affect the type/amount of student learning as well as policies and 
programs that may affect these intervenors. Six intervenors are 
covered: student involvement , learning expectations , assessment and 
feedback ^indiyicJuaiizat ion , inst^ methods, and content. The 

intervenors are used as criteria to review the quality of 
instructional programs^ Three types of impr strategies are 

considered: student input (e^g^ , admissions , financial aid) , ^.rocess 
(resources and curriculum) , and outcomes .Examples of quality 
improvement strategies for each type of strategy are identified. 
Finally, six impiications f or inst^tut i focusing on 

student learning and local improyement efforts are identi f ied , 
including: preparation and continuing education needs of 
practitioners, and a hew role and focus for policy analysis. (SW) 
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ABSTRACT 

Institutional researchers are urged to return to activities 
which focus more directly on the fundamental Issues of teaching 
and learning, to return to the "heart of the matter". Major 
purposes of the paper are: 1) to familiarize practitioners with 
key concepts from theory arid research on student learning and to 
identify some of the important literature available; 2) to 
provide a conceptual and analytical framework for use with the 
design of instructional improvement efforts at the local level; 
and, 3) to identify and discuss some of the implications this 
shift in focus has for institutional research. Development of 
the framework was motivated by a desire to promote more 
comprehensive, multiple and less "quick fix" approaches to the 
improvement of instructional program quality. The framework 
identifies six factors^ or "in tervenor s" , that appear to have 
consequences for the kind and amount of student learning taking 
place and policies and programs, or "improvement strategies", 
that hold promise for affecting the intervenors. 
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iSTRdDUCTION 

Factors largely external to educatidii have resulted in 
renewed national attention on issues of "quality: How effective 
(or ineffective) are our schools and colleges? and. How can 
they be improved? One of the morie significant of these factors 
is the growing concern over whether our citizenry will be 
prepared to deal with future economic, social and technological 
change^ Education is increasingly viewed as the critical 
vehicle for economic development and international economic 
positioning • 

The K-12 sector has responded with "school improvement" 
efforts at the school, district and statewide levels. These 
efforts have been shaped in large part by the research on 
school effectiveness and place emphasis on fostering conditions 
designed to increase student learning. Higher education is 
only beginning to shape its response, one that can be expected 
to accelerate over the next several years. 

Institutional research has historically played an important 
support role in efforts to improve nstruc tional programs 
(Dressel, 1980; Saupe, 1981; Norris, 1983). For the past 
decade, however, bur profession has focused heavily on 
activities related to matters of efficiency, enrollment^ 
retention arid planning and the development of computer-based 
data systems designed to support these and other planning and 
management activities. Our profession has often seemed fixated 
on numerical and computerized digits and widgets! 

The point here is not to devalue or to suggest the 
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abandonment of work in these areas but to urge a return by the 
institutional research community to activities that focus more 
directly on the fundamental issues of teaching and learning. 
The call here is for a return to "the heart of the 
matter "--improving student learning. 

The major purposes of this paper are: 1) to familiarize 
institutional research practitioners with key concepts from 
theory and research on student learning and to identify some of 
the important literature available; 2) to provide a conceptual 
and analytical framework for use with the design of 
instruc*-ional improvement efforts at the local level; and, 3) 
to identify and discuss some of the implications this shift in 
focus has for institutional researchi 

Development of the framework, initially supported by 
Management Services, Inc. of the National Center for Higher 
EducF.tion Management Systems, was motivated by a desire to 
prbjibte more comprehensive, multiple and less "quick fix" 
ppproaches to the improvement of instructional program 
quality. Many institutions appear far too ready to grab hold 
of one particular strategy in an effort to improve "quality" 
overnight. Popular examples include: raising admission 
requirements; initiating new marketing programs; adopting new 
testing practices; and the large-scale purchase of 
microcomputers. 

The framework identifies factors, here called "intervenor s" , 
that appear to have direct or indirect consequences for the 
kind and amount of student learning that takes place. Also 
identified are various policies and programs, labeled 
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"improvement strategies", which hold promise for affecting 
these ihtervenors. 

THE CONCEPTUAL AND ANALYTICAL FRAMEWORK 

The conceptual framework, shown in Figure 1, suggests the 
following: 1) improving instructional program quality should be 
defined as improving student learning; 2) a series of factors — 
"Intervenors" — can be identified which theory, research and 
experience suggest influence the kind and amount of student 
learning that takes place; and 3) a set of strategies^ policy 
and programmatic, can be identified which hold varying degrees 
of promise for affecting the "interveners", and, consequently, 
student learning. Following a discussion of each of the 
framework components, examples of using the framework to shape 
a local quality improvement effort are presented. 

The Imp foy erne nt Frame vork : An Overview 
Figure 1 

Strategy Types Intervenors 

Student Input 
Resources 
Curriculum 
Outcomes 

Defining "Quality" as the Improvement of Student Learning 

Critical to the framework is the definition of "quality" as 
the improvement of student learning. Astin (1982) provides 
several def ihit ions of "quality", including those based on 
program or institutional reputation and resources. He urges 



Involvement K 

Expectations V Student 

Assessment /Feedback , y Learning 

Individualization y/ 
Instructional Methods 
Content 
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adoption of an alternative definition — talent development 
(1985, p. 61): 

In its simplest terms the talent development 
conception of excellence focuses on changes 
in the student from the beginning to the end 
of an educational program. These changes can 
cover a wide rang of cognitive and affective 
attributes • 

Astin advocates a definition which requires information on 
changes or improvements in student performance over time. The 
effectiveness of our institutions and programs is to be judged 
in terms of the difference they make in both the kind and 
amount of student learning that takes place after initial 
institutional entry. A strong appeal of the sttident learning 
definition is that it focuses attention on the use rather than 
the acquisLtij^fi. of resources (Astin, 1982) and on the 
fundamental mission of education: the development of human 
resources . 

The specific definition of "quality" adopted here, then, is: 
"change of particular kinds" in students between institutional 
entry and exit, or, student learning over time. Importantly, 
this definition has both an empirical and normative dimension. 
Both are critical for an institution to address. The empirical 
dimension focuses only on change and asks: "Did change occur?"; 
"How much?" and, "In which direction?" It is essentially a 
measurement issue. The normative dimension focuses on 
improvement and asks: "Did the desired change occur?" The 
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normative, unlika the empirical dimension, requires a focus on 
values (i.e., What will be the viiued student learning?) as 
well as measurement (i.e., Did the valued learning take place? 
How much?). 

The first steps in an improvident effort, then, are to define 
the valued student learning and to establish procedures and 
measures to help assess the kind and amount of learning taking 
place. Ewell (1984) provides one of the iore useful 
classifications of student outcomes (see Table 1). These 
outcomes may be defined for a particular course, instructional 
program or the entire undergraduate or graduate educational 
experience . 

A eiassificat xdti of Outicoines Dxnensions 
Table 1 

li Knowledge Outcomes 

a. General Knowledge (Breadth of Kridwledge) 

b. Knowledge of Specific Fields (Depth of Knowledge) 

2. Skills Outcomes 

a. General Competence (Social Functioning) Skills 

b . Professional/Occupational Skills 

3. Attitude/Value Outcomes 

a. Personal Goals and Aspiratidris 

b. General Attitudes, Values and Satisfactions 

c. Attitudes Toward Self (Development of Identify) 

d. Attitudes Toward Others 

4. Relationshipswith Society and with Particular 
Constituencies 

a. Relationships with Educational Institutions 

b. Relationships with Employers/Industries 
Ci Relationships with Professions 

d. Relationships, with Family /Community /Society 

Source; Ewell (1984). The Seli^^giar di^g- X^^itu t ion . 
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The "InterTenors": Factors Affecting Student Learning 

A review of school effectiveness research (for example, see 
Brookover, et.al., 1982; Cuban, 1984; and National eommittee 
for Citizens in Educatidh, 1980) > recent higher education 
reports such as Iftv^vement in Learning (1984) and pedagogical 
theory (see Gagne, 1977; Chickering and Associates, 1981; and 
Astih, 1985) suggests six intervening variables^ or conditions 
for learning, which appear related to the kind and amount of 
student learning that takes place. This list and definitions 
presented in Table 2 should be considered preliminary and are 



The liiteryendrs 
Table 2 



intervenor 



Definition 



Student Involvement: 



Learning Expectations: 



Assessment and Feedback: 



Individualization: 



instructional Methods: 



Content : 



the time, energy, investment and/or 
effort students put into their own 
learning experience 

expected levels of _ perf ormance on 
specific kinds of learning outcomes 

regular and periodic assessment of 
student learning; monitoring progress 
and feedback to both students and 
instructors 

the taking into account of the 
individual differences of learners 
when designing and implementing 
learning programs 

the instruct ional strategies used 
to f acilitate/support/br ing about 
student achievement of the desired 
learning outcomes 

the "stuff" or subject matter used 
to facilitate the valued learning; 
what students spend their time doing 
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offered as a pbiht of departure in deisighihg an instructional 
improvement plan. The intervenorjg may be used as criteria to 
evaluate current instructional programs as well as the 
potential value of alternative instructional program 
improvement strategies. The framework assumes that each 
interverior may be shaped or "manipulated" by institutional 
policies and programs. 

For Astin (1985) ^ chief spokesperson for a student 
involvement theory^ involvement means "the amount of physical 
and psychdldgical energy that the student devotes to the 
academic experience "( p . 134 ) j such as preparing for a course 
exam or involvement in a more general set of experiences. He 
asserts that "the effectiveness of any educational policy or 
practice is directly related to the capacity of that policy or 
practice to increase student involvement (p.l36)i 

The Xjj^alvement in Learning report (National Institute of 
Education, 1984), in addition to discussing student 
involvement, identifies two other conditions critical to 
student learning: 1) the extent to which expectations about 
what is to be learned and the level of performance to be 
achieved are made clear and are communicated publically; and 2) 
the extent to which regular and periodic assessment and 
feedback takes place. Studentis^ faculty and administrators all 
should be recipients of such assessment. 

Especially interesting is the closeness of match between 
findings from the school effectiveness research and the 
J nvolve fflent in Learjilftg report. Both define an "excellent" or 
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quality program (or school) as one where students are 
learning. Both identify "expectations" and "feedback" as 
critical factors. Both cali for greater attention to how 
students spend their time. In the K-12 sector this is often 
called "time on task" or "academic learning time" while in 
higher education^ the current term is "involvement in 
learning". Another similarity is that neithei provides 
cookbook-like solutions. Rather^ each advocates the 
implementation of strategies that take into account local 
problems, strengths and resources. 

indi vidualiza 'on is another critical intervenor^ The 
assumption made is that we do not all learn equally effectively 
in the same ways. Chicker ing ( 1981 ) ^ and Cross(1981), among 
others, are proponents of the importance of taking individual 
differences into account in the design of instructional 
programs. Attention is given to issues of curricular content 
and instructional methods that best meet the needs of an 
individual student. Advocates of self-paced instruction, 
contract learning, independent study^ a large role for 
electives arid competency-based learning models are associated 
with this iridividualized approach. Emerging work in the areas 
of "learning styles", gender roles arid multicultural education 
also have impdrtarit implicat ibris here. 

Instructional methods, ariother intervenor^ could be subsumed 
under the factors of studerit involvement , expectations and 
assessment and feedback; that is* methods would be evaluated in 
terms of the extent to which they promote student involvement, 
clearly articulate learning expectations and the extent to 
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which use is made of assessment and feedback. It is included 
as the fifth intervenor, however, to underscore the importance 
of including a review of instructional methods as part of an 
improvement effort and identifying the extent to which teachers 
are using methods that appear to hold the most promise for 
promoting particular kinds of learning (see McKeachie, 1978; 
and Sherman, 1985). 

The sixth intervenor, content, refers to the issues or the 
subject matter around which learning experiences are 
organized. What should students be reading? How should they 
be spending their time? What should they be encouraged to 
think about and to do? Content Has to do with Issues of the 
intrinsic worth of an activity as well as its potency for 
bringing about the desired learning. 

ReTieving Current Instructional Prbgrans 

Prior to reviewing alternative strategies which might become 
part of a comprehensive improvement effort, the iiitervenbrs 
should be used as criteria to review the quality of current 
instructional programs (see Table 3). Such a review can 
identify the kinds of changes that appear to be most needed. 

It is important to note that this review process focuses 
directly on the iiitervenbrs thismselves and nbt on the 
measurement bf student learning outcomes. While assessment 
information on learning outcomes would be useful to have, it is 
not essential to initiating an imprbvement effort. In fact, 
direct infbrmatibn about the iriterveriors is the more important 
data jset in terms of identifying problem areas. Assessment 
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information, while important, by itself tells little about the 
steps that should be taken to improve student learning. The 
interveriors provide such an improvement road map. 



Criteria for Use In Revieviiig 
Current Instructional Prograns 

table 3 



1 • Involvement ; To what extent are students encouraged to be 
involved in their own learning experiences? How much time are 
they involved? How active are they in the learning process? Do 
current institutional and program policies promote student 
involvement? 

2» Expectations : How clear andknown tdstudents are the 
expected levels of performance ? In particular classes? In major 
programs? For the undergraduate or graduate experience as a 
whole? 

3. Assessment ! Is student learning regularly and 
periodically assessed , student progress monitored and feedback 
provided to s tudehts i faculty and administrators? Does 
assessement activity fulfill only an accountability role (i.e., 
are certain standards being met or levels of performance being 
achieved?) or also an improvement role (i.e., what progress are 
students making? how much learning has taken place in the lower 
division experience?) 

4. Individualization ; To what extent and how are individual 
differences among students taken into account in courses and 
programs? Is use made of testing and placement procedures? Are 
individualized learning opportunities available? How do 
instructors deal with student diversity of various kinds in their 
classrooms and _ programs? How prepared do they feel to deal with 
such diversity? 

5. Instructional Methods ; What kinds of instructional 
strategies and approaches are used by faculty? To what extent are 
active modes of learning encouraged ? How well matched are the 
methods With the expected learning outcomes and with the 
content/subject matter? Do the methods reflect current research 
findings on effectiveness? 

6. Content/Sub ifect Matter ; Is the subject matter aligned 

clo sely to the expected learning outcomes? Are students spenJihg 
sufficient time to achieve the desired outcomes? Is the material 
itself intrinsically of value? 
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The Isprovesent Strategies: Pblicj and Prt^fgrnmnatic 
Levers for Change 

Three major imprbvement strategy types are idisntified and 
labeled: Student Inputs Throughput or Process (Resources and 
Curriculum) and Outcomes (see Figure 2). These should be viewed 
as opportunities for policy or prdgrammatic change with 
potential for influencing the intervenoras and, consequieiitly , 
student learning. An effective improve me nt program would 
include the use of a combination of specific strategies drawn 
from each of these three types. 

The laproveaent FraBevork Expanded 
Figure 2 



RESOURCES: information. Personnel, Facilities, Dollars 



STUDENT 



STRATEGICS 



involvement 
Expectations 
Assessmr^nt and Feedback 
Indfvieual Izatlon 
h?etbods 
Cvintent 

(INTIRVEHORS) 



CURRICULUM: Contert, Methods, Organization 



TWOUGI^^ OR PROCESS STRATEGIES 



OUTCOMES 
KhoWl edge 
Skills 
Attitudes 



STUDENT 



OUTCOME 
STRATEGIES 



Examples of each of the tihree strategy types are shown in 
Table 4, A local improvement effort will need to develop its 
own list by asking the question: What kinds of policy or 
programmatic changes can be made in student input, resources, 
the curriculum or oi:tcbmes which hold the most promise for 
affecting one or more of the intervenors? 
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Quality laproTeaent Strategies: 
Ezaaples from the Field 

Table 4 



Strategy-Type 



Examples 



Student. Input; 
Changing Incoming 
Student Characteristics 



Resources: 

Chariging the Amount and 
Quality of Information. 
Personnel, Physical and 
Fiscal Resources 



Curriculum; - 
Changing the Cearning 
Content and How 
tearning.Experlences 
are Organized 



Outcomes. 

Changing the Bases 
on Which Learning is 
Assessed and 
Certified 



1. 
2. 
3. 
4. 
5. 
6. 
2. 
8. 
9. 

Id. 
li. 

12. 
13. 
14. 
15. 

I. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 

1. 
2. 
3. 
4. 
5. 
6. 



admissions.. . 
financial _a1d 

cpllaborat1on_with_ schools 

high school graduation requirements 

marketing 

better Information about the school for 
prospective students 



self-study/prbgram review 
outcomes infbrmatlbh- 
dtaghbstlc testing/placement 
advising 

faculty rscfutttnent/hiring 

promottoH^^tenare _ __ . . 

faculty;^ staff dovelopment 

Instructional evaluation 

merit pay 

technplpgy 

financial aid 

physical plant 

library 

student housing 

total dollar acquisition 

general education core requirements 
creation of "learning communities" 
interdiscipl ihary approaches 
competency^based approaches 
experlencedrased. approaches 
oew.prQgram/course.guidelines 
instructional . methods. 
stud.e.nt-teacher_rat1ps_ 
'calendar and scheduling 

achievement testing required to graduate 

grading practices 

graduate warranty (cite OSU) 

licensure requirements 

certificatibri requiremehs 

policies for admissibh to particular 

courses, programs, lovels of study 



Developing Strategies for Local Use 

A local instructional program impfovement committee or task 
force would examine systematically, not mechanistically, the 
improvement strategies which hold the most promise for 
affecting the intervenors. No "blueprint" is available for 
doing this. An improvement plan must be designed for a 
particular local setting and situation by the leadership most 
familiar with that setting. The discussion to follow 
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demonstrates the kind of conceptual and analytical framew5rk a 
local improvement team would bring to the task of designing a 
comprehensive instructional program improveient effort. 

Input Strategies refer to policies and prog^iis designed to 
change the characteristics of the entering student population. 
Prior learning expectations or requirements might be clarified 
and better communicated to prospective applicants, thus 
attracting students better qualified to participate 
successfully in the programs offered. Collaborative efforts 
among the schools, two- and fbUr-year institutions could 
include opportunities for students to learn earlier and prior 
to admission how well prepared they are. Financial aid 
policies f:ould be examined in terms of their implications for 
student involvement (ie.g. Should greater effort be made to 
increase the numbers of students participating full-time, thus 
providing them with greater opportunities for program 
involvement?) 

Many Resource Strategies are available, strategies designed 
to alter the amount and quality of informational, personnel, 
physical and fiscal resources available. Only three are 
discussed briefly here: program review; the faculty; and 
technology^ Program review, a vehicle for increasing the 
quality and use of information, is a widely used planning and 
management process in higher education (Barak^ 1982; Conrad and 
Wilson, 1985)i A program review effort could be expanded to 
include information about the interveners. The process offers 
an opportunity for clarifying learning expectations ^ the 
content and instructional methods to be used and the 
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appropriateness of each and the extent to which the individual 
differences of students are taken into account in program and 
instructional design. Program review can serve as a 
comprehensive improvement strategy allowing for the examination 
of all of the intervenbrs. 

Anothier major Resource Strategy involves examining policies 
and practices affecting the faculty^ their hiring, promotion, 
tenure^ post-tenure review and development. Some of the 
questions that could be raised include: Is teaching truly 
valued^ as reflected in promotion and tenure policies and 
actions taken? Is there time and are workloads such that 
students have access to faculty? Do faculty have strong 
classroom and program assessment skills? How skilled are 
faculty with using alternative instructional methods? Two 
recent, provocative books on faculty that are well worth 
consulting are The American Academic Pr of isssibn (Finkelstein , 
1984) and Improvinig Undergraduate Education Throujgh Faculty 
Developm e nt (McKeachie and Eble, 1985). 

Increased access to computers is viewed by many as essential 
to future institutional viability. For some, computers are a 
new indicator of program quality* The computer, along with 
other liearhihg technologies such as video and audio, has been 
prbmdted as a tobl fbr use in increasing instructional 
r espbhsiveness to the individual differences bf student 
learherst including differences in ability^ learning styles and 
motivation (Gilbert and Green, 1986; and Lewis, 1985), 

Potential Curriculum Strategies are also plehti.ful^ those 
designed to change the content and organization of learning 

18 
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sxperiencesi Many institutions have completed of are in the 
process of examining their general education core or redefining 
"liberal education". Such an examination provides an 
opportunity, like program review, for considering implications 
for all of the intervenors; Which learning outcomes do we wish 
to promotie and how? How will student involvement be promoted? 
How will we know if students are making progress in thie desired 
directions? .vand so forthi Excellent resources include: Gamson 
(1984)^ with her notion of "learning communities"; Alverno 
College Faculty (1976); and, Adelman (1984). 

Lastly, Outcome Strategies are those designed to change how 
learning is assessed and certified. The improvement task force 
could ask: What kinds of feedback do students receive and when 
oh the progress they are making? During the course of each 
term or only at the end? Should standards beyond the 
completion of courses be met to advance to upper division 
standing? Teacher education graduates from Oregon State 
University are covered by a "graduate warranty", a kind of 
performance jguarantee backed by the promise of assistance to 
those experiencing difficulties as new classroom teachers^ 
Outcome strategies hold promise for affecting the expectations 
and assessment and feedback intervenbrsi One excellent source 
of information about the use of outcome strategies- is the 
American Association for Higher Education in Washington, DiC, 
Order the set of papers prepared for the Oc ber 1985 National 
Conference oh Assessment in Higher Educatioi. eld in Columbia, 
South Carolina. (Also see Ewell, 1985 and Pace, 1979.) 

o 
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IMPLICATIONS FOR INSTITUTIONAL RESEARCH 

The shift In focus to student learning and to local 
improvement efforts designed to promote student learning has 
several major implications for Institutional research. Six are 
identified and discussed. 

Structural Realignment 

To be effectively involved in instructional improvement 
efforts, institutional researchers must be linked closely with 
the academic affairs or instructional part of the organization 
and enjoy credibility with the faculty. They should be 
included on local instructional improvement teams or task 
forces. They must have these linkages in order to understand 
and be responsive to the information needs of their faculty 
colleagues ; 

Preparation and Continuing Education Needs of Practitioners 

Professionals entering the field need to have greater- 
knowledge of instructional theory and research, curriculum 
design and evaluation, educational assessment and measurement 
and, in general, a broader understanding of the current theory 
and research base that has implications for student learningi 
The field needs to attract more individuals with backgrounds in 
disciplines such as anthropology, psychology, sociology, and 
political science who are also familiar with the literature of 
higher education. 

o 
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A Nev Sole and Focas for Policy Analysis 

Practitioners need to develop more knowledge of and skill 
with policy analysis^ The focus of such analyses should 
include the evaluation of alternative policies in terms of the 
potential they hold for promoting the kinds of student learning 
valued by the institution * The policy arena needs to be 
expanded beyond issues of access to include issues of student 
learning end developments 

The Developaent of Nev Assessment Tools 

A quick review of the intervenors and their use as part of an 
instructional program review process suggests the need for new 
instruments or tools of assessment. Several sources previously 
cited (Harris, 1985; Ewell^ 1985) identify a number of such 
instruments currently available, including an important new 
instrument developed by C. Robert Pace (1984) for assessing the 
kind and level of student involvement taking place. More 
instruments are needed to describe the local situation with 
respect to other intervenors and valued student learning 
outcomes • 

The Local Institutional Research Agenda 

The local institutional research agenda should include: 1) 
projects related more directly to understanding conditions 
which appear to promote and improve student learning; 2) 
projects designed to monitor and analyze overtime the impact of 
new policy and programmatic efforts to improve student 

21 
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learning; and 3) collaborative research projects involving 
discipline-based as well as Education faculty and their 
students • 

Mew Linkages with Local School District Researchers 

For too long the school and postsecohdary research 
communities have been separated, implicitly suggesting that 
they have nothing to learn or to gain from one another. Any 
postsecondary type who begins to review the school 
effectiveness and related literature will quickly come to the 
opposite conclusion, that much is to be gained from crossing 
educational boundaries ^ Institutional researchers in colleges 
and universities can gain much from Initiating new linkages 
with those doing research in school settings. 

CONCLUDING COMMENT 

This paper began and ends with a call for a return by 
institutional research practitioners to ah arena referred to 
here as the "heart of the matter" — student learhingi 
Instructional program improvement efforts heed the conceptual, 
analytical and technical support our profession can providei 
The issues are too urgent for us to sit back and wait for the 
direction and leadership of others. We need to provide our own 
leadership and become advocates for program improvement efforts 
that are designied to increase the student learning valued by 
our institutions. 
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